Systems and methods for facilitating a transaction using augmented reality

ABSTRACT

A system includes one or more memory devices storing instructions, and one or more processors configured to execute the instructions to perform steps of a method. The system may provide an augmented environment that facilitates a transaction. The system may store profile data including user payment or user profile information. The system may then receive environmental data, and identify one or more action items in the environmental data. In response to this identification, the system may augment the environmental data by adding virtual environmental data, and then provide this virtual environmental data to a device to create an augmented environment. The system can then receive user input data, and provide purchase request data to a merchant terminal in response to those inputs.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of, and claims priority under 35U.S.C. § 120 to, U.S. patent application Ser. No. 15/664,749, filed Jul.31, 2017, which claims the benefit of U.S. Provisional Application No.62/437,699, filed Dec. 22, 2016, the entire contents of which are fullyincorporated herein by reference.

FIELD OF INVENTION

The present disclosure relates to systems and methods for facilitating atransaction using augmented reality, and more particularly using aninteractive augmented environment.

BACKGROUND

Often, while moving through the world, people see things that remindthem of transactions they would like to or need to complete. Forexample, a person may be opening mail and be reminded of a cable billdue. As another example, a person may be looking in a cabinet orrefrigerator and be made aware of or reminded that an item is runninglow or absent. Typically, in situations such as these, a person needs toremember the task they wish to complete, and then go to a store or to acomputer terminal connected to an appropriate merchant.

However, it is not always easy to keep track of these tasks, nor is italways practical to go to a store. It is quite common for a person toforget the listing of items they intended to purchase, or to put offpaying a bill and forgetting it in a pile of junk mail. At present thereare some methods and applications such as grocery subscription servicesor task reminders to help address these issues; however they stillrequire time to set-up and monitor.

Accordingly, there is a need for systems and methods for facilitating atransaction using augmented reality. Such systems can be non-invasive toan individual's day and can provide the ability to complete transactionsinstantaneously. Embodiments of the present disclosure are directed tothis and other considerations.

SUMMARY

Disclosed embodiments provide systems and methods for facilitating atransaction using augmented reality.

Consistent with the disclosed embodiments, the system may include one ormore memory devices storing instructions, and one or more processorsconfigured to execute the instructions to perform steps of a method tofacilitate a transaction using augmented reality. The system may storeprofile data including user payment or other profile information. Thesystem may then receive environmental data, and identify one or moreaction items in the environmental data. In response to thisidentification, the system may augment the environmental data by addingvirtual environmental data, and then provide this virtual environmentaldata to a device. The system can then receive user inputs, and providepurchase request data to a merchant terminal in response to thoseinputs.

Consistent with the disclosed embodiments, methods for providing anaugmented environment that facilitates a transaction are also provided.

Further features of the disclosed design, and the advantages offeredthereby, are explained in greater detail hereinafter with reference tospecific embodiments illustrated in the accompanying drawings, whereinlike elements are indicated by like reference designators.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference will now be made to the accompanying drawings, which are notnecessarily drawn to scale, and which are incorporated into andconstitute a portion of this disclosure, illustrate variousimplementations and aspects of the disclosed technology and, togetherwith the description, serve to explain the principles of the disclosedtechnology. In the drawings:

FIG. 1 is a diagram of an exemplary system that may be used to generatean augmented environment and facilitate a transaction;

FIG. 2 is a component diagram of an exemplary service provider terminal;

FIG. 3 is a flowchart of an exemplary method for generating an augmentedenvironment and facilitating a transaction;

FIG. 4 is a diagram of exemplary types of virtual environmental data;and

FIG. 5 is a component diagram of an exemplary computing device.

DETAILED DESCRIPTION

Some implementations of the disclosed technology will be described morefully with reference to the accompanying drawings. This disclosedtechnology may, however, be embodied in many different forms and shouldnot be construed as limited to the implementations set forth herein. Thecomponents described hereinafter as making up various elements of thedisclosed technology are intended to be illustrative and notrestrictive. Many suitable components that would perform the same orsimilar functions as components described herein are intended to beembraced within the scope of the disclosed electronic devices andmethods. Such other components not described herein may include, but arenot limited to, for example, components developed after development ofthe disclosed technology.

It is also to be understood that the mention of one or more method stepsdoes not preclude the presence of additional method steps or interveningmethod steps between those steps expressly identified. Similarly, it isalso to be understood that the mention of one or more components in adevice or system does not preclude the presence of additional componentsor intervening components between those components expressly identified.

The disclosed embodiments are directed to systems and methods forgenerating an augmented environment and facilitating a transaction. Thesystem may include one or more memory devices storing instructions, andone or more processors configured to execute the instructions to performsteps of a method. Specifically, in some embodiments, the system mayprovide an augmented environment that facilitates a transaction. Toaccomplish this, the system may execute the instructions to storeprofile data including user payment or other profile information. Thesystem may then receive environmental data, and identify one or moreaction items in the environmental data. In response to thisidentification, the system may augment the environmental data by addingvirtual environmental data, and then provide this virtual environmentaldata to a device. Having provided the virtual data to the computingdevice, the system may then receive user inputs in response to thevirtual environment. With this user input data, the system can providepurchase request data to a merchant terminal to complete a transaction.

In another embodiment, a system for facilitating a transaction mayinclude a memory for storing profile data comprising user paymentcredentials and user profile information. The system may include anenvironmental data sensor for obtaining environmental data comprisingaudio and/or visual information representing a physical environment. Thesystem may include an input/output device for receiving virtualenvironmental data comprising data associated with a merchant from acomputing device. The system may further include an augmentedenvironment processor that, based on environmental data obtained fromthe environmental data sensor, may identify one or more action items inthe environmental data, augment the environmental data by adding thevirtual environmental data comprising enhanced content associated withone or more of the one or more action items, and generate a signalcomprising augmented environmental data. Enhanced content may be contentthat is unrepresented in the physical environment. The system mayfurther include an augmented environmental data display that may receivethe signal comprising augmented environmental data and display saidaugmented environmental data in accordance with said signal and a userinterface device for receiving user input data associated with one ormore action items.

In another aspect, a non-transitory computer-readable medium storinginstructions is disclosed. The instructions, when executed by one ormore processors, may cause a computing device to perform steps of amethod for generating an augmented environment and facilitating atransaction. Specifically, the computing device may store profile dataincluding user payment or other profile information and receiveenvironmental data. The computing device may then identify one or moreaction items in the environmental data. This identification can resultin the system augmenting the environmental data by adding virtualenvironmental data, and then providing this virtual environmental datato a device. The system may then provide purchase request data to amerchant terminal to complete a transaction in response to user inputsreceived from the device.

In another aspect, a method of facilitating a transaction is disclosed.The disclosed method can include storing profile data comprising userpayment credentials and user profile information, and receivingenvironmental data from a computing device associated with the storedprofile data comprising audio and visual information representing aphysical environment. Next, the method can include identifying one ormore action items in the environmental data, and augmenting theenvironmental data by adding virtual environmental data comprisingenhanced content associated with one or more of the one or more actionitems. The method can further include providing the virtualenvironmental data to the computing device to create an augmentedenvironment in combination with the physical environment. The method canthen include receiving user input data associated with one or more ofthe action items from the computing device, and then providing purchaserequest data to a merchant terminal to enable a transaction related tothe one or more action items.

Reference will now be made in detail to exemplary embodiments of thedisclosed technology, examples of which are illustrated in theaccompanying drawings and disclosed herein. Wherever convenient, thesame references numbers will be used throughout the drawings to refer tothe same or like parts.

FIG. 1 is a diagram of an exemplary system that may be configured toperform one or more processes that can generate interactive virtualelements to augment an environment. The components and arrangementsshown in FIG. 1 are not intended to limit the disclosed embodiments asthe components used to implement the disclosed processes and featuresmay vary.

In accordance with disclosed embodiments, transaction facilitatingsystem 100 may include a service provider terminal 110 in communicationwith a computing device 120 via network 130. In some embodiments,service provider terminal 110 is also in communication with a merchantdatabase terminal 140 via network 130. Computing device 120 may includea mobile computing device (e.g., a smart phone, tablet computer, smartwearable device, portable laptop computer, voice command device, orother mobile computing device) or a stationary computing device (e.g., astationary desktop or other stationary computer). Computing device 120may belong to or be provided by a consumer, or may be borrowed, rented,or shared. In some embodiments, computing device 120 may be configuredto provide one or more of an augmented reality and a virtual realityexperience to one or more users, and may be equipped with a visualdisplay, speakers or other auditory devices, tactile simulators, andother devices configured to engage the user's senses to enhance theaugmented/virtual reality experience.

In some embodiments, transaction facilitating system 100 may connect toa merchant database terminal 140 that communicates with other devices ofsystem 100 via network 130. Optionally, in some embodiments, interactivevirtual/augmented environment system 100 may also include a merchantpoint of sale (POS) terminal 150 that communicates with other devices ofsystem 100 via network 130. Additionally, in some embodiments,interactive virtual/augmented environment system 100 may also include afinancial service provider terminal 160 that communicates with otherdevices of system 100 via network 130.

Network 130 may be of any suitable type, including individualconnections via the internet such as cellular or WiFi networks. In someembodiments, network 130 may connect terminals using direct connectionssuch as radio-frequency identification (RFID), near-field communication(NFC), Bluetooth™, low-energy Bluetooth™ (BLE), WiFi™, ZigBee™ ambientbackscatter communications (ABC) protocols, USB, or LAN. Because theinformation transmitted may be personal or confidential, securityconcerns may dictate one or more of these types of connections beencrypted or otherwise secured. In some embodiments, however, theinformation being transmitted may be less personal, and therefore thenetwork connections may be selected for convenience over security.

An exemplary embodiment of service provider terminal 110 is shown inmore detail in FIG. 2 . Computing device 120, merchant database terminal140, merchant POS terminal 150, and financial service provider terminal160 may have a similar structure and components that are similar tothose described with respect to service provider terminal 110. As shown,service provider terminal 110 may include a processor 210, aninput/output (“I/O”) device 220, a memory 230 containing an operatingsystem (“OS”) 240 and a program 250. For example, service providerterminal 110 may be a single server or may be configured as adistributed computer system including multiple servers or computers thatinteroperate to perform one or more of the processes and functionalitiesassociated with the disclosed embodiments. In some embodiments, theservice provider terminal 110 may further include a peripheralinterface, a transceiver, a mobile network interface in communicationwith the processor 210, a bus configured to facilitate communicationbetween the various components of the service provider terminal 110, anda power source configured to power one or more components of the serviceprovider terminal 110.

A peripheral interface may include the hardware, firmware and/orsoftware that enables communication with various peripheral devices,such as media drives (e.g., magnetic disk, solid state, or optical diskdrives), other processing devices, or any other input source used inconnection with the instant techniques. In some embodiments, aperipheral interface may include a serial port, a parallel port, ageneral purpose input and output (GPIO) port, a game port, a universalserial bus (USB), a micro-USB port, a high definition multimedia (HDMI)port, a video port, an audio port, a Bluetooth™ port, a near-fieldcommunication (NFC) port, another like communication interface, or anycombination thereof.

In some embodiments, a transceiver may be configured to communicate withcompatible devices and ID tags when they are within a predeterminedrange. A transceiver may be compatible with one or more of:radio-frequency identification (RFID), near-field communication (NFC),Bluetooth™, low-energy Bluetooth™ (BLE), WiFi™, ZigBee™ ambientbackscatter communications (ABC) protocols or similar technologies.

A mobile network interface may provide access to a cellular network, theInternet, or another wide-area network. In some embodiments, a mobilenetwork interface may include hardware, firmware, and/or software thatallows the processor(s) 210 to communicate with other devices via wiredor wireless networks, whether local or wide area, private or public, asknown in the art. A power source may be configured to provide anappropriate alternating current (AC) or direct current (DC) to powercomponents.

Processor 210 may include one or more of a microprocessor,microcontroller, digital signal processor, co-processor or the like orcombinations thereof capable of executing stored instructions andoperating upon stored data. Memory 230 may include, in someimplementations, one or more suitable types of memory (e.g. such asvolatile or non-volatile memory, random access memory (RAM), read onlymemory (ROM), programmable read-only memory (PROM), erasableprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), magnetic disks, optical disks,floppy disks, hard disks, removable cartridges, flash memory, aredundant array of independent disks (RAID), and the like), for storingfiles including an operating system, application programs (including,for example, a web browser application, a widget or gadget engine, andor other applications, as necessary), executable instructions and data.In one embodiment, the processing techniques described herein areimplemented as a combination of executable instructions and data withinthe memory 230.

Processor 210 may be one or more known processing devices, such as amicroprocessor from the Pentium™ family manufactured by Intel™ or theTurion™ family manufactured by AMD™ Processor 210 may constitute asingle core or multiple core processor that executes parallel processessimultaneously. For example, processor 210 may be a single coreprocessor that is configured with virtual processing technologies. Incertain embodiments, processor 210 may use logical processors tosimultaneously execute and control multiple processes. Processor 210 mayimplement virtual machine technologies, or other similar knowntechnologies to provide the ability to execute, control, run,manipulate, store, etc. multiple software processes, applications,programs, etc. One of ordinary skill in the art would understand thatother types of processor arrangements could be implemented that providefor the capabilities disclosed herein.

Service provider terminal 110 may include one or more storage devicesconfigured to store information used by processor 210 (or othercomponents) to perform certain functions related to the disclosedembodiments. In one example, service provider terminal 110 may includememory 230 that includes instructions to enable processor 210 to executeone or more applications, such as server applications, networkcommunication processes, and any other type of application or softwareknown to be available on computer systems. Alternatively, theinstructions, application programs, etc. may be stored in an externalstorage or available from a memory over a network. The one or morestorage devices may be a volatile or non-volatile, magnetic,semiconductor, tape, optical, removable, non-removable, or other type ofstorage device or tangible computer-readable medium.

In one embodiment, service provider terminal 110 may include memory 230that includes instructions that, when executed by processor 210, performone or more processes consistent with the functionalities disclosedherein. Methods, systems, and articles of manufacture consistent withdisclosed embodiments are not limited to separate programs or computersconfigured to perform dedicated tasks. For example, service providerterminal 110 may include memory 230 that may include one or moreprograms 250 to perform one or more functions of the disclosedembodiments. Moreover, processor 210 may execute one or more programs250 located remotely from system 100. For example, system 100 may accessone or more remote programs 250, that, when executed, perform functionsrelated to disclosed embodiments.

Memory 230 may include one or more memory devices that store data andinstructions used to perform one or more features of the disclosedembodiments. Memory 230 may also include any combination of one or moredatabases controlled by memory controller devices (e.g., server(s),etc.) or software, such as document management systems, Microsoft™ SQLdatabases, SharePoint™ databases, Oracle™ databases, Sybase™ databases,or other relational databases. Memory 230 may include softwarecomponents that, when executed by processor 210, perform one or moreprocesses consistent with the disclosed embodiments. In someembodiments, memory 230 may include a merchant information database 260,an environment database 270, and a virtual environmental database 280for storing related data to enable service provider terminal 110 toperform one or more of the processes and functionalities associated withthe disclosed embodiments. Merchant information database 260 may includestored data relating to product price, product inventory, relatedproducts, and available discounts associated with one or more productsassociated with one or more merchants. Environment database 270 mayinclude stored data relating to a physical environment that is capturedby one or more devices (e.g., computing device 120) of system 100.Virtual environment database 280 may include stored data relating to avirtual environment that is provided to and can be updated based uponinteractions with one or more devices (e.g., computing device 120) ofsystem 100.

Service provider terminal 110 may also be communicatively connected toone or more memory devices (e.g., databases (not shown)) locally orthrough a network. The remote memory devices may be configured to storeinformation and may be accessed and/or managed by service providerterminal 110. By way of example, the remote memory devices may bedocument management systems, Microsoft™ SQL database, SharePoint™databases, Oracle™ databases, Sybase™ databases, or other relationaldatabases. Systems and methods consistent with disclosed embodiments,however, are not limited to separate databases or even to the use of adatabase.

Service provider terminal 110 may also include one or more I/O devices220 that may comprise one or more interfaces for receiving signals orinput from devices and providing signals or output to one or moredevices that allow data to be received and/or transmitted by serviceprovider terminal 110. For example, service provider terminal 110 mayinclude interface components, which may provide interfaces to one ormore input devices, such as one or more keyboards, mouse devices, touchscreens, track pads, trackballs, scroll wheels, digital cameras,microphones, sensors, and the like, that enable service providerterminal 110 to receive data from one or more users (such as viacomputing device 120).

In exemplary embodiments of the disclosed technology, service providerterminal 110 may include any number of hardware and/or softwareapplications that are executed to facilitate any of the operations. Theone or more I/O interfaces may be utilized to receive or collect dataand/or user instructions from a wide variety of input devices. Receiveddata may be processed by one or more computer processors as desired invarious implementations of the disclosed technology and/or stored in oneor more memory devices.

While service provider terminal 110 has been described as one form forimplementing the techniques described herein, those having ordinaryskill in the art will appreciate that other, functionally equivalenttechniques may be employed. For example, as known in the art, some orall of the functionality implemented via executable instructions mayalso be implemented using firmware and/or hardware devices such asapplication specific integrated circuits (ASICs), programmable logicarrays, state machines, etc. Furthermore, other implementations of theterminal 110 may include a greater or lesser number of components thanthose illustrated.

Merchant database terminal 140 may have one or more processors 142 andone or more merchant databases 144, which may be any suitable repositoryof merchant data. Merchant database terminal 140 may be located at thePOS location, off-site at another merchant location, or at a third partylocation. Information stored in merchant database terminal 140 may beaccessed (e.g., retrieved, updated, and added to) via network 130 by oneor more devices (e.g., service provider terminal 110) of system 100. Inother embodiments, merchant POS terminal 150 may be configured toprocess online transactions on behalf of the associated merchant.Merchant database 144 may store information relating to products andservices offered by merchants such as pricing, quantity, availability,discounts, reviews, and any other such generally available informationthat a consumer may utilize in making a purchasing decision. In someembodiments, merchant database 144 may also include location informationassociated with products and services that identifies the location(s)that a particular product or service is available for purchase. In someembodiments, the location information may include an identification of aparticular store, terminal, or kiosk that the product or service may bepurchased from.

Merchant POS terminal 150 may have one or more POS devices 152, 154, 156that communicate with one or more devices (e.g., computing device 120)of system 100 via network 130. In some embodiments, POS devices 152,154, 156 may be associated with one or more products or items at a POSlocation, and may serve as beacons to identify the associated productsor items and related information to one or more devices of system 100.

FIG. 5 shows an exemplary embodiment of computing device 120. As shown,computing device 120 may include input/output (“I/O”) device 220 forreceiving virtual environmental data comprising data associated with amerchant from a computing device, memory 230 containing operating system(“OS”) 240 and program 250 and all associated components as describedabove with respect to service provider terminal 110. Computing device120 may also have one or more processors, including an augmentedenvironment processor 502 for generating signals representative ofaugmented and/or virtual environments, a geographic location sensor(“GLS”) 504 for determining the geographic location of computing device120 and/or an associated environmental data sensor 508, an augmentedenvironmental data display 506 for displaying augmented environmentaldata, an environmental data (“ED”) sensor 508 for obtainingenvironmental data including audio and/or visual informationrepresenting a physical environment, and a user interface (“U/I”) device510 for receiving user input data associated with one or more actionitems. According to some embodiments, a user interface device 510 mayinclude some or all of the components described with respect toinput/output device 220 above. In some embodiments, environmental datasensor 508 may include a microphone or a digital camera. According tosome embodiments, augmented environment processor 502 may include all ofthe features and functions of processor 210 described above. Further,augmented environment processor 502 may be configured to, based onenvironmental data obtained from the environmental data sensor, identifyone or more action items in the environmental data, augment theenvironmental data by adding the virtual environmental data comprisingenhanced content associated with one or more of the one or more actionitems, and generate a signal comprising augmented environmental data.Enhanced content may be content that is unrepresented in the physicalenvironment. The augmented environmental data display 506 may beconfigured to receive the signal comprising augmented environmental dataand display the augmented environmental data in accordance with thesignal.

According to some embodiments, based on the user input data, the one ormore action items, and the user profile information, the input/outputdevice 220 may be configured to provide purchase request data to amerchant terminal 150 to enable a transaction related to the one or moreaction items, wherein the purchase request data comprises the storeduser payment credentials. In some embodiments, receiving user input datamay include detecting one or more of a visual command and an auditorycommand in the user input data and selecting, based on the detection,one or more of the merchant and the set of user credentials.

FIG. 3 shows a flowchart of method 300 for generating an augmentedenvironment and facilitating an action. Method 300 may be performed byservice provider terminal 110 using processor 210 to execute memory 230.In some embodiments, one or more steps of method 300 may be delegated toother devices or systems in system 100, such as computing device 120,merchant database terminal 140, merchant POS terminal 150 and/orfinancial service provider terminal 160. Following method 300, user(s)may generate, share, and modify virtually augmented environments.

In block 310, service provider terminal 110 may store profile data.Profile data may include information such as payment credentials and/oruser profile information. Profile data may be entered directly by a user(e.g., via computing device 120 or merchant POS terminal 150), or storedon and retrieved from merchant database terminal 140 or financialservice provider terminal 160. Payment credentials may include creditcard information, including standard or tokenized versions of thecredentials, financial account information (e.g., which may beassociated with financial service provider terminal 160) or otherfinancial information as needed to initiate and authorize a payment,while profile information can include delivery locations, billingaddresses, previous transactions, other transaction metadata (e.g.,location, time, etc.), user preferences for clothing or other productsizes and quantities, and merchant account numbers.

At step 320, service provider terminal 110 may receive environmentaldata comprising audio and visual information via computing device 120.Having received the environmental data, at step 330 service providerterminal 110 may identify one or more action items in the visualinformation of the environmental data. Action items can be identified byrecognizing shapes (e.g., via 2D or 3D image recognition), bar codes,RFID devices, QR codes, IR-, BLE-, and other beacons located in anenvironment (e.g., POS devices 152, 154, and 156 associated with one ormore products or items at a POS location or devices associated withdifferent predefined areas of a closet, refrigerator, pantry, etc. thatstores items that are periodically repurchased), or the like. Examplesof action items may include products to be re-ordered, bills to be paid,objects related to bills to be paid or products to be ordered, or thelike.

At step 340, service provider terminal 110 may augment the environmentaldata by adding virtual environmental data. In some embodiments, thisaugmentation may involve displaying information from a merchant (e.g.,who may be associated with merchant database terminal 140 and/ormerchant POS terminal 150) that is specific to one of the one or more ofthe identified action items. For example and not limitation, serviceprovider terminal 110 may recognize a particular item or product andaugment the environment with visual effects such as a price tag or billamount, bill due date, availability of the product and/or products ofother brands or sizes, or aesthetic effects such as virtual fireworks,sale signs, colors indicating past due bills or bills due within 24hours, and the like. Service provider terminal 110 can connect to anappropriate merchant database terminal 140 to include informationrelevant to the identified action item. For example, in the event thatthe identified action item is a cable box, service provider terminal 110may connect to a cable provider database terminal (e.g., merchantdatabase terminal 140), or in the event that the action item is a fooditem, service provider terminal 110 may connect to a grocery storedatabase terminal (e.g., merchant database terminal 140). In someembodiments, the augmented environmental data may include inventoryinformation or amount information associated with one or more actionitems. In some embodiments, the service provider terminal 110 maydetermine, from the environmental data, the number or amount of aproduct in the environment. For example, the service provider terminalmay determine from image analysis that a bottle of ketchup is halfempty, or that a box of soda cans only has four cans left. In someembodiments, inventory information or amount information may be gatheredfrom various environmental sensors such as scanners (e.g., barcode orRFID scanners), weight sensors, temperature sensors, or any other typeof suitable sensor.

In block 350, service provider terminal 110 may provide the virtualenvironmental data to computing device 120 in order to create anaugmented environment when viewed in combination with the physicalenvironment. In some embodiments, the virtual environment data mayinclude some or all of the environmental data provided by computingdevice 120. In some embodiments, computing device 120 may only receivethe newly generated virtual environmental data. The virtualenvironmental data may include product price or bill amount information,bill due date, product inventory information, related products (e.g.,other brands, sizes, or models), available discounts, and aestheticvisual effects (e.g., target graphics, glow, highlighted colors, orfirework graphics displayed around one or more products to drawattention to a particular item or to convey a message associate withthat item, such as being highlighted red for a past due bill). In someembodiments, the augmented environment may display inventory informationor amount information in association with a particular item. Forexample, computing device 120 may display that there are four cans ofsoda left in association with a box of soda cans. According to someembodiments, the service provider terminal 110 may receive datarepresented of predefined desired inventory or amount levels. Forexample, a user may specify a minimum inventory level of four soda cansin association with a box of soda cans, such that when the system 100detects that there are less than four soda cans, the service providerterminal 110 may generate a prompt to buy more soda.

After displaying the augmented environment to a user, computing device120 may receive user input in response to that augmented environment,and this user input can be communicated to other devices in system 100(e.g., service provider terminal 110). For example, the user ofcomputing device 120 may touch a screen, make a gesture in the visualfield of a video capturing component of computing device 120, speak acommand into an audio capturing component of computing device 120, enteran input using a connected input device, or some combination of these.In some embodiments, computing device 120 may be a virtual realitydevice including a headset and a camera. In such embodiments, the usermay be viewing the augmented environment, and may physically reach foran element in that environment to generate user input data. In someembodiments, computing device 120 can be a personal computer, with theaugmented environment displayed on a monitor. A user may then select(e.g., using the mouse or other controller, or via motion if a motionsensor or device with an accelerometer is used in cooperation with thecomputer) a virtual element in the augmented environment to generateuser input data. The user input data may then be transmitted to serviceprovider terminal 110 (e.g., by computer device 120 via network 130).

In response to receiving the user input data at block 360, at block 370,service provider terminal 110 may provide transaction request data tothe merchant database terminal 140 or merchant POS terminal 150 in orderto enable a transaction. In some embodiments, the user input datacorresponds with a selection of a product for purchase at a merchant. Insome embodiments, the user input data may include a selection of apayment method for a bill or invoice. For example, the user input datamay be the user gesturing to press a virtual “reorder” button associatedwith a food item, and service provider terminal 110 connects to agrocery merchant terminal and provides payment information to completethe transaction with the grocery merchant to order the food item.Similarly, the user input data may be the user gesturing to press avirtual “pay bill” button associated with a cable bill, and serviceprovider terminal 110 connects to a cable merchant terminal and providespayment information to pay the identified bill with the cable merchant.

In some embodiments, to provide the transaction request data, serviceprovider terminal 110 may transmit a purchase authorization request tofinancial service provider terminal 160, which may already have useraccount information associated with computing device 120. Once therequest has been authorized, service provider terminal 100 may transmita purchase request to merchant POS terminal 150 to purchase theidentified product(s) or to pay the identified bill(s). Alternatively,in other embodiments, service provider terminal 100 may add theidentified products or identified bills to a checkout cart associatedwith computing device 120 for purchase or payment at another time.

Optionally, in some embodiments, system 100 may facilitate anotheraction in lieu of or in addition to making a purchase or paying a bill.For example, system 100 may allow a user to access and interact with(e.g., via computing device 120) one or more billing statements,purchase histories, or other account information associated with theaction item identified in block 330, and this account information may beincluded in the virtual environment data that is provided to computingdevice 120 at block 350. For example, as a user observes a credit card,system 100 may identify the credit card as an action item, and providevirtual environmental data that includes information related to paymenthistory or a previous statement to computing device 120 to create anaugmented environment. Then, via computing device 120, system 100 mayreceive user input data and, in response, provide action request data toa merchant terminal (e.g., merchant POS terminal 150). Some examples ofactions may include saving an item to a shopping list, sharing an itemwith a friend via a social connection, displaying item information suchas ratings or reviews for the item or similar items, and requestingtransaction financing such as a merchant credit account or the like.

FIG. 4 shows a diagram that represents an embodiment of exemplary typesof virtual environmental data 400. For example, virtual environmentaldata 400 may include product information 410, related productinformation 420, product availability between merchants 430, usercredential options 440, or aesthetic visual effects 450. Productinformation 410 can include, for example, a product price, stockinginformation, product features, and product reviews. Related productinformation 420 may include complementary products (e.g. batteries for adevice, matching tie for a shirt, etc.) or alternative products (e.g.value-sized items, more highly rated alternatives, comparable brands,etc.). Merchant information 430 can include prices, shipping costs, andavailability at various merchants, as well as merchant reviews. Usercredential options 440 can include individual family member profiles(e.g. this is for mom, dad, or child), credit cards, membership numbers(e.g. loyalty programs, frequent flier accounts, etc.), bank accountinformation, or merchant account passwords. Examples of aesthetic visualeffects may include product image graphics, glowing effects,highlighting objects with colors, or animations (e.g. firework graphics,waving flags, etc.).

As used in this application, the terms “component,” “module,” “system”and the like are intended to include a computer-related entity, such asbut not limited to hardware, firmware, a combination of hardware andsoftware, software, or software in execution. For example, a componentmay be, but is not limited to being, a process running on a processor,an object, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on acomputing device and the computing device can be a component. One ormore components can reside within a process and/or thread of executionand a component may be localized on one computer and/or distributedbetween two or more computers. In addition, these components can executefrom various computer readable media having various data structuresstored thereon. The components may communicate by way of local and/orremote processes such as in accordance with a signal having one or moredata packets, such as data from one component interacting with anothercomponent in a local system, distributed system, and/or across a networksuch as the Internet with other systems by way of the signal.

Certain embodiments and implementations of the disclosed technology aredescribed above with reference to block and flow diagrams of systems andmethods and/or computer program products according to exampleembodiments or implementations of the disclosed technology. It will beunderstood that one or more blocks of the block diagrams and flowdiagrams, and combinations of blocks in the block diagrams and flowdiagrams, respectively, can be implemented by computer-executableprogram instructions. Likewise, some blocks of the block diagrams andflow diagrams may not necessarily need to be performed in the orderpresented, may be repeated, or may not necessarily need to be performedat all, according to some embodiments or implementations of thedisclosed technology.

These computer-executable program instructions may be loaded onto ageneral-purpose computer, a special-purpose computer, a processor, orother programmable data processing apparatus to produce a particularmachine, such that the instructions that execute on the computer,processor, or other programmable data processing apparatus create meansfor implementing one or more functions specified in the flow diagramblock or blocks. These computer program instructions may also be storedin a computer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meansthat implement one or more functions specified in the flow diagram blockor blocks.

As an example, embodiments or implementations of the disclosedtechnology may provide for a computer program product, including acomputer-usable medium having a computer-readable program code orprogram instructions embodied therein, said computer-readable programcode adapted to be executed to implement one or more functions specifiedin the flow diagram block or blocks. Likewise, the computer programinstructions may be loaded onto a computer or other programmable dataprocessing apparatus to cause a series of operational elements or stepsto be performed on the computer or other programmable apparatus toproduce a computer-implemented process such that the instructions thatexecute on the computer or other programmable apparatus provide elementsor steps for implementing the functions specified in the flow diagramblock or blocks.

Accordingly, blocks of the block diagrams and flow diagrams supportcombinations of means for performing the specified functions,combinations of elements or steps for performing the specifiedfunctions, and program instruction means for performing the specifiedfunctions. It will also be understood that each block of the blockdiagrams and flow diagrams, and combinations of blocks in the blockdiagrams and flow diagrams, can be implemented by special-purpose,hardware-based computer systems that perform the specified functions,elements or steps, or combinations of special-purpose hardware andcomputer instructions.

Certain implementations of the disclosed technology are described abovewith reference to mobile computing devices. Those skilled in the artrecognize that there are several categories of mobile devices, generallyknown as portable computing devices that can run on batteries but arenot usually classified as laptops. For example, mobile devices caninclude, but are not limited to portable computers, tablet PCs, internettablets, PDAs, ultra mobile PCs (UMPCs), wearable devices, and smartphones. Additionally, implementations of the disclosed technology can beutilized with internet of things (IoT) devices, smart televisions andmedia devices, appliances, automobiles, toys, and voice command devices,along with peripherals that interface with these devices.

In this description, numerous specific details have been set forth. Itis to be understood, however, that implementations of the disclosedtechnology may be practiced without these specific details. In otherinstances, well-known methods, structures and techniques have not beenshown in detail in order not to obscure an understanding of thisdescription. References to “one embodiment,” “an embodiment,” “someembodiments,” “example embodiment,” “various embodiments,” “oneimplementation,” “an implementation,” “example implementation,” “variousimplementations,” “some implementations,” etc., indicate that theimplementation(s) of the disclosed technology so described may include aparticular feature, structure, or characteristic, but not everyimplementation necessarily includes the particular feature, structure,or characteristic. Further, repeated use of the phrase “in oneimplementation” does not necessarily refer to the same implementation,although it may.

Throughout the specification and the claims, the following terms take atleast the meanings explicitly associated herein, unless the contextclearly dictates otherwise. The term “connected” means that onefunction, feature, structure, or characteristic is directly joined to orin communication with another function, feature, structure, orcharacteristic. The term “coupled” means that one function, feature,structure, or characteristic is directly or indirectly joined to or incommunication with another function, feature, structure, orcharacteristic. The term “or” is intended to mean an inclusive “or.”Further, the terms “a,” “an,” and “the” are intended to mean one or moreunless specified otherwise or clear from the context to be directed to asingular form. By “comprising” or “containing” or “including” is meantthat at least the named element, or method step is present in article ormethod, but does not exclude the presence of other elements or methodsteps, even if the other such elements or method steps have the samefunction as what is named.

While certain embodiments of this disclosure have been described inconnection with what is presently considered to be the most practicaland various embodiments, it is to be understood that this disclosure isnot to be limited to the disclosed embodiments, but on the contrary, isintended to cover various modifications and equivalent arrangementsincluded within the scope of the appended claims. Although specificterms are employed herein, they are used in a generic and descriptivesense only and not for purposes of limitation.

This written description uses examples to disclose certain embodimentsof the technology and also to enable any person skilled in the art topractice certain embodiments of this technology, including making andusing any apparatuses or systems and performing any incorporatedmethods. The patentable scope of certain embodiments of the technologyis defined in the claims, and may include other examples that occur tothose skilled in the art. Such other examples are intended to be withinthe scope of the claims if they have structural elements that do notdiffer from the literal language of the claims, or if they includeequivalent structural elements with insubstantial differences from theliteral language of the claims.

Exemplary Use Cases

The following exemplary use case describes one example of a typical userflow pattern. It is intended solely for explanatory purposes and not inlimitation. The user may have credit card information and a deliveryaddress stored in the system (e.g., system 100), and may be in theirhome looking into their kitchen cabinets and refrigerator using theirmobile device (e.g., computing device 120). The augmented environmentseen by the user (e.g., via computing device 120) may include a seriesof purchasing options to reorder items in the cabinets and refrigeratorthat may be empty or near empty. For example, the system may visuallyidentify the contents of the container to be below a certain threshold,or it may connect with sensors either in the containers themselves or innearby accessories (e.g., fridge), that may sense the weight or fullnessof a particular container. The system may include predefined minimuminventory levels or product amounts that may be specified by the user.For example, a user may specify that a predefined minimum number of cansof soup, such as four cans, so that when the system detects that theuser has less than four cans of soup in their cupboard, the system(e.g., via computing device 120) may present a prompt to buy more soup.The augmented environment may also provide (e.g., via computing device120) suggestions of complementary items to complete recipes. Usingcomputing device 120, the user can tap the virtual items to add them toa virtual shopping cart, and then swipe the virtual cart to checkout andorder the items in the cart.

Another exemplary use case relates to a bill payment system, whereby auser can use a mobile device (e.g., computing device 120) to view aparticular set of items in their home to bring up bill information andapply payments to their accounts. For example, the user could view alamp in their home to view electric bill information retrieved by thesystem from a utility database, and pay the bill with a swipe or tap. Inthis manner, system 100 could receive environmental data including thelamp (e.g., via computing device 120), identify the lamp as an actionitem (e.g., via service provider terminal 110), retrieve relatedelectric bill information from merchant database terminal 140, andprovide an augmented environment (e.g., via computing device 120) thatdisplays electric bill information to the user. Then, upon receivinguser input data via computing device 120, service provider terminal 110may provide a transaction request (e.g., to merchant POS terminal 150)to pay the identified bill. Similarly, the user could then view theircable box to view cable billing information and make payments.Additionally, the user could use the system when opening bills receivedvia email or postal mail, and view additional information in theaugmented environment that allows for the scheduling of payment,reminders, or the like. Such a system may allow a user to ensure allbills are paid in a timely manner perhaps by simply remembering to scantheir home with their mobile device once or twice a month. The systemmay also automatically recognize when bills are entering the household,e.g., by scanning or recognizing them at the mailbox, or by scanningelectronic mailboxes, e.g., email inbox for incoming bills.

Another exemplary use case relates to using a mobile device (e.g.,computing device 120) to view an augmented environment that presents theuser with inventory information relating to products being viewed by theuser (e.g., products viewed by a camera of computing device 120). Asdescribed above, the system may identify an object, such as a box ofsoda cans in a refrigerator, and the system may determine the inventoryof the object, by for example, visually identifying the number of cansin the box or by determining the number of cans present using a weightsensor. Then, every time the user opens the door to their refrigerator,the system (e.g., via computing device 120) may display an augmentedenvironment that shows, in association with the box of soda, that thereare, for example, only four cans of soda remaining in the box. In thisway, the system (e.g., system 100) may enable the user to passivelymonitor the inventory or amount of particular items.

Certain implementations of the disclosed technology are described abovewith reference to block and flow diagrams of systems and methods and/orcomputer program products according to example implementations of thedisclosed technology. It will be understood that one or more blocks ofthe block diagrams and flow diagrams, and combinations of blocks in theblock diagrams and flow diagrams, respectively, can be implemented bycomputer-executable program instructions. Likewise, some blocks of theblock diagrams and flow diagrams may not necessarily need to beperformed in the order presented, may be repeated, or may notnecessarily need to be performed at all, according to someimplementations of the disclosed technology.

These computer-executable program instructions may be loaded onto ageneral-purpose computer, a special-purpose computer, a processor, orother programmable data processing apparatus to produce a particularmachine, such that the instructions that execute on the computer,processor, or other programmable data processing apparatus create meansfor implementing one or more functions specified in the flow diagramblock or blocks. These computer program instructions may also be storedin a computer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meansthat implement one or more functions specified in the flow diagram blockor blocks. As an example, implementations of the disclosed technologymay provide for a computer program product, including a computer-usablemedium having a computer-readable program code or program instructionsembodied therein, said computer-readable program code adapted to beexecuted to implement one or more functions specified in the flowdiagram block or blocks. Likewise, the computer program instructions maybe loaded onto a computer or other programmable data processingapparatus to cause a series of operational elements or steps to beperformed on the computer or other programmable apparatus to produce acomputer-implemented process such that the instructions that execute onthe computer or other programmable apparatus provide elements or stepsfor implementing the functions specified in the flow diagram block orblocks.

As used herein, unless otherwise specified the use of the ordinaladjectives “first,” “second,” “third,” etc., to describe a commonobject, merely indicate that different instances of like objects arebeing referred to, and are not intended to imply that the objects sodescribed must be in a given sequence, either temporally, spatially, inranking, or in any other manner.

What is claimed is:
 1. A system, comprising: one or more processors; anda memory in communication with the one or more processors and storinginstructions that, when executed by the one or more processors, areconfigured to cause the system to: receive, from an environmental datasensor, environmental data comprising visual information representing aphysical environment, the environmental data comprising a gesture madein a visual field of the environmental data sensor; detect, based on theenvironmental data obtained from the environmental data sensor, one ormore purchased products and a number of the one or more purchasedproducts in the environmental data by identifying one or more objectspresent in the visual information using 3D image recognition techniques;determine that the number of the one or more purchased products is belowa predetermined threshold; augment the environmental data by addingvirtual environmental data comprising enhanced content associated withthe one or more purchased products that comprises a prompt to buy moreof the one or more purchased products; generate a signal comprising theaugmented environmental data; display, using a display, the augmentedenvironmental data in accordance with the signal; receive user inputdata comprising a visual command to select a merchant based on thegesture made in the visual field of the environmental data sensor; andprovide, based on the user input data, purchase request data to amerchant terminal associated with the merchant to enable a transactionrelated to the one or more purchased products.
 2. The system of claim 1,wherein the prompt to buy more of the one or more purchased productscontains a list of merchants, wherein the user input data includes aselection of the merchant from the list of merchants.
 3. The system ofclaim 1, wherein the one or more purchased products are one or morecanned items.
 4. The system of claim 1, wherein one or more weightsensors, one or more volume sensors, the 3D image recognitiontechniques, or a combination thereof are used to detect the number ofthe one or more purchased products.
 5. The system of claim 1, whereinthe system is a mobile phone, a virtual reality headset, or a personalcomputer.
 6. A system, comprising: one or more processors; and a memoryin communication with the one or more processors and storinginstructions that, when executed by the one or more processors, areconfigured to cause the system to: receive, from a computing device,environmental data comprising visual information representing a physicalenvironment; detect one or more purchased products and a number of theone or more purchased products in the environmental data by identifyingone or more objects present in the visual information using 3D imagerecognition techniques; determine that the number of the one or morepurchased products present in the environmental data is below apredetermined threshold; augment the environmental data by addingvirtual environmental data comprising enhanced content associated withone or more of the one or more purchased products, wherein the enhancedcontent comprises a prompt to buy more of the one or more purchasedproducts; provide, to the computing device, the virtual environmentaldata to create an augmented environment in combination with the physicalenvironment; receive user input data associated with one or more of thepurchased products from the computing device, wherein receiving the userinput data comprises: detecting a visual command in the environmentaldata, the visual command comprising a gesture made in the visual fieldof a video capturing component of the computing device; and selecting,based on the detection of the visual command, a merchant; and provide,based on the user input data and the one or more purchased products,purchase request data to a merchant terminal associated with themerchant to enable a transaction related to the one or more purchasedproducts, wherein the purchase request data comprises stored userpayment credentials.
 7. The system of claim 6, wherein the virtualenvironmental data comprises product price information, product stockinformation, related products, available merchants, transaction options,user credential options, aesthetic visual effects, or a combinationthereof.
 8. The system of claim 6, wherein the computing devicecomprises a mobile phone, a virtual reality headset, or a personalcomputer.
 9. The system of claim 6, wherein the one or more processorsare further configured to execute the instructions to select themerchant terminal based on the one or more objects present in the visualinformation.
 10. The system of claim 6, wherein a set of usercredentials is selected based on the one or more objects present in thevisual information.
 11. The system of claim 10, wherein one of the oneor more purchased products is a representative object that is related tothe set of user credentials.
 12. The system of claim 6, whereindetecting the one or more purchased products further comprisesidentifying visual indicia in the visual information, and wherein thevisual indicia comprises a text string, a symbol, a group of symbols, aUPC code, or a QR code.
 13. The system of claim 6, wherein the one ormore purchased products are canned items.
 14. A non-transitorycomputer-readable medium storing instructions that, when executed by oneor more processors, cause a computer system to: receive, from acomputing device associated with stored profile data, environmental datacomprising visual information representing a physical environment;detect one or more purchased products and a number of the one or morepurchase products in the environmental data by identifying one or moreobjects present in the visual information using 3D image recognitiontechniques; determine that the number of the one or more purchasedproducts in the environmental data is below a predetermined threshold;augment the environmental data by adding virtual environmental datacomprising enhanced content associated with one or more of the one ormore purchased products, wherein the enhanced content comprises a promptto buy more of the one or more purchased products; provide, to thecomputing device based on detecting the number of the one or morepurchased products, the virtual environmental data to create anaugmented environment in combination with the physical environment;receive user input data associated with one or more of the purchasedproducts from the computing device, wherein receiving the user inputdata comprises: detecting a visual command in the environmental data,the visual command comprising a gesture made in a visual field of avideo capturing component of the computing device; and selecting, basedon the detection of the visual command, a merchant; and provide, basedon the user input data, the one or more purchased products, and storeduser profile information, purchase request data to a merchant terminalassociated with the merchant to enable a transaction related to the oneor more purchased products, wherein the purchase request data comprisesstored user payment credentials.
 15. The non-transitorycomputer-readable medium of claim 14, wherein the virtual environmentaldata comprises product price information, product stock information,related products, available merchants, transaction options, usercredential options, aesthetic visual effects, or a combination thereof.16. The non-transitory computer-readable medium of claim 14, whereindetecting the one or more purchased products further comprisesidentifying visual indicia in the visual information, and wherein thevisual indicia comprises a text string, a symbol, a group of symbols, aUPC code, or a QR code.
 17. The non-transitory computer-readable mediumof claim 14, wherein the one or more purchased products are one or morecanned items.
 18. The non-transitory computer-readable medium of claim14, wherein the instructions are further configured to cause thecomputer system to select the merchant terminal based on the one or moreobjects present in the visual information.
 19. The non-transitorycomputer-readable medium of claim 14, wherein the computing devicecomprises a mobile phone, a virtual reality headset, or a personalcomputer.